Statistical Induction, Severe Testing, and Model Validation

A number of important methodological issues in statistical modeling and inference depend crucially on the notion of statistical induction adopted. An attempt is made to articulate the notion of statistical induction underlying modern frequentist inference going back to Fisher (1922). The paper brings out the differences in the nature of inductive inference associated with estimation and prediction on one hand and testing on the other; the former based on factual and and the latter on counterfactual reasoning. Induction by enumeration is placed in a formal statistical context in order to bring out its crucial weaknesses. Particular emphasis is placed on the role of pre-data type I and II error probabilities, as measures of the ‘trustworthiness’ of test procedures. Post-data, error probabilities can be used to render the traditional coarse accept/reject decision more informative by evaluating the severity with which a hypothesis or a claim passes a particular test, with data x. The discussion emphasizes the nature of the severity assessment and the associated post-data error probabilities, as they relate to the pre-data error probabilities. Supplementing N-P testing with the severity evaluation gives rise to the error-statistical account of inference which constitutes the most complete description of frequentist statistical induction. The evaluation of error probabilities (pre-data and post-data) assumes the validity of the statistical premises, because any departures will render the inductive inference unreliable by creating a divergence between the nominal and actual error probabilies. The paper discusses the importance of ensuring statistical adequacy using thorough misspecification testing and respecification. It also demonstrates how statistical adequacy can be used to shed light on a number of methodological problems such as model validation vs. model selection and statistical vs. substantive adequacy. ∗Section 3 of the paper relies heavily on joint work with Deborah Mayo.